feat(pip): Add a clean-sheet pip implementation #650

arrdem · 2025-09-25T18:26:22Z

An implementation of pip based on consuming PEP-751 [1] like lockfiles. Specifically uv lockfiles, which contain internal dependency graph information that the PEP-751 specification labels optional.

Follows in the footsteps of rules_js's pnpm support by consuming a lockfile which contains enough information to produce materialize dependencies without performing any repository time operations which could be platform dependent.

Features

Supports cross-platform builds of wheels
Supports hermetic source builds of wheels
Automatically handles dependency cycles
Creates unified pip hubs which span virtualenv/dependency solution boundaries
Pip library targets from deactivated venvs are incompatible
Platform constraints on pip libraries do not prevent the creation of a library target

Example

# MODULE.bazel content
pip = use_extensioin("@aspect_rules_py//pip:extesion.bzl", "pip")
pip.declare_hub(hub_name = "pip")
    
pip.declare_venv(hub_name = "pip", venv_name = "a")
pip.lockfile(hub_name = "pip", venv_name = "a", lockfile = "third_party/py/venvs/pylock-a.toml")

pip.declare_venv(hub_name = "pip", venv_name = "b")
pip.lockfile(hub_name = "pip", venv_name = "b", lockfile = "third_party/py/venvs/pylock-b.toml")

use_repo(pip, "pip")

# BUILD.bazel content

py_venv_binary(
  name = "foo",
  srcs = [
    "foo.py",
  ],
  main = "foo.py",
  deps = [
    "@pip//cowsay", # Pull cowsay from the configured venv
  ],
  venv = "a", # Configure the default venv to be "a"; may be overriden at the CLI
)

The active venv state can be overriden at the cli by specifying --'@pip//venv=b' here for instance, or by using transitions to(re) set that same flag.

Appendix

[1] https://peps.python.org/pep-0751/
[2] https://peps.python.org/pep-0751/#locking-build-requirements-for-sdists

Changes are visible to end-users: yes

Searched for relevant documentation and updated as needed: yes/no
Breaking change (forces users to change their own code or config): yes/no
Suggested release notes appear below: yes/no

To do list

Test plan

Manually test flipping the venv command line flag
Manually test flipping the venv transition attr
Create py_venv_tests covering that different versions of the same package can be concurrently configured via different venvs
Create a py_venv_test covering that Airflow or another package with dependency cycles can be provisioned
Create a py_venv_binary embedded in and transitioned for a Linux OCI container across arch boundaries

aspect-workflows · 2025-09-25T19:56:45Z

Test

⚠️ Buildkite build #464 failed.

//py/tests/py_venv_conflict:test_venv_ignore failed to build

Errors encountered while applying Starlark transition

Errors encountered while applying Starlark transition

//docs:update_5_test failed to build

in fail_with_message_test rule //docs:update_5_test:
Traceback (most recent call last):
	File
"/mnt/ephemeral/output/rules_py/__main__/external/aspect_bazel_lib/lib/private/fail_with_message_test.bzl",
line 4, column 9, in _fail_with_message_test_impl
		fail(ctx.attr.message)
Error in fail:
 
@//docs:unstable.md does not exist. To create & update this and other generated files, run:
 
    bazel run @//docs:update
 
To create an update *only* this file, run:
 
    bazel run //docs:update_5

@@pypi_setuptools//:whl failed to build

no such package '@@pypi_setuptools//': The repository '@@pypi_setuptools' could not be resolved: Repository
'@@pypi_setuptools' is not defined

@@pip2//click:click failed to build

no such package '@@pip2//click': The repository '@@pip2' could not be resolved: Repository '@@pip2' is not
defined

7 other actions failed to build.

Failed tests (14)

//:requirements_test [k8-fastbuild]                                                       🔗
//docs:update_6_test [k8-fastbuild]                                                       🔗
//examples/multi_version:py_version_default_test [k8-fastbuild]                           🔗
//examples/multi_version:py_version_test [k8-fastbuild-ST-494921797612]                   🔗
//py/tests/py-binary:runfiles_from_pip_test [k8-fastbuild]                                🔗
//py/tests/py-external-venv:test [k8-fastbuild-ST-4d16e4d42f67]                           🔗
//py/tests/py_image_layer:my_app_layers_test_test [k8-fastbuild]                          🔗
//py/tests/py_venv_conflict:validate_import_roots [k8-fastbuild-ST-02ecc3a0f3d8]          🔗
//py/tests/py_venv_image_layer:my_app_amd64_layers_test [k8-fastbuild]                    🔗
//py/tests/py_venv_image_layer:my_app_arm64_layers_test [k8-fastbuild]                    🔗
//py/tests/py_venv_image_layer:py_amd64_image_command_test [k8-fastbuild]                 🔗
//py/tests/py_venv_image_layer:py_amd64_image_content_test [k8-fastbuild]                 🔗
//py/tests/py_venv_image_layer:py_arm64_image_content_test [k8-fastbuild]                 🔗
//py/tests/virtual/django:requirements_test [k8-fastbuild]                                🔗

💡 To reproduce the build failures, run

bazel build //py/tests/py_venv_conflict:test_venv_ignore //docs:update_5_test @@pypi_setuptools//:whl @@pip2//click:click

💡 To reproduce the test failures, run

bazel test //py/tests/py_venv_image_layer:py_arm64_image_content_test //py/tests/py_venv_image_layer:py_amd64_image_command_test //py/tests/py-external-venv:test //py/tests/py_venv_image_layer:my_app_amd64_layers_test //py/tests/py-binary:runfiles_from_pip_test //py/tests/py_venv_image_layer:py_amd64_image_content_test //py/tests/virtual/django:requirements_test //examples/multi_version:py_version_default_test //examples/multi_version:py_version_test //py/tests/py_venv_conflict:validate_import_roots //:requirements_test //py/tests/py_image_layer:my_app_layers_test_test //docs:update_6_test //py/tests/py_venv_image_layer:my_app_arm64_layers_test

Still need to plumb in toolchains & real installer.

pip/private/sdist_build/rule.bzl

dzbarsky · 2025-09-29T23:57:02Z

pip/private/constraints/abi/macro.bzl

+
+            for minor in MINORS:
+                selects.config_setting_group(
+                    name = "is_{}{}{}".format(interpreter, major, minor),


generally %s is faster than .format because it avoids a function call and has simpler (faster) arg conversion. Also if you have only a single substitution it helps even more because you don't even need to construct a tuple. Up to you where to use it, but at least for the functions in the extension that are called repeatedly I would strongly consider it. (Honestly I pretty much always use it unless I have a complex/multiline template with 4+ args and/or using an arg multiple times)

dzbarsky

since you said you love nits...i left a bunch of nits. feel free to apply or disregard as you see fit!

dzbarsky · 2025-09-29T23:57:52Z

pip/private/constraints/defs.bzl

+    # We loop up to the second-to-last item to ensure we always have a 'next' stage.
+    for i, current_stage in enumerate(stages):
+        selects.config_setting_group(
+            name = "{}".format(current_stage.name),


nit: name = current_stage.name :)

dzbarsky · 2025-09-29T23:58:33Z

pip/private/constraints/defs.bzl

+        selects.config_setting_group(
+            name = "{}".format(current_stage.name),
+            match_any = [
+                ":{}".format(current_stage.condition),


":" + current_Stage.condition or ":%s" % current_stage.condition are both faster and less chars :)

dzbarsky · 2025-09-30T00:01:47Z

pip/private/constraints/repository.bzl

+"""
+selects.config_setting_group(
+    name = "{}",
+    match_all = {},


it might be a bit clearer to put the brackets in the expression here, especially since I think the way you did it I think the trailing ] isn't indented correctly? Here's a similar thing I did with build_deps here

dzbarsky · 2025-09-30T00:04:07Z

pip/private/extension.bzl

+    # Collect all hubs, ensure we have no dupes
+    for mod in module_ctx.modules:
+        for hub in mod.tags.declare_hub:
+            hub_specs.setdefault(hub.hub_name, {})


I saw you use setdefault like this a bunch, I think you can tweak the pattern slightly to avoid repeated lookups:

hub_specs.setdefault(hub.hub_name, {})[mod.name] = 1

dzbarsky · 2025-09-30T00:07:04Z

pip/private/extension.bzl

+    problems = []
+    for hub_name, modules in hub_specs.items():
+        if len(modules.keys()) > 1:
+            problems.append(


I think you could combine this with the previous loop and make hub_specs store hub_name -> mod_name(just check if you already have an entry and add toproblems) instead of of hub_name -> mod_name -> 1`. I guess it would make it harder to handle 3 modules all using the same name, but meh? It would probably lead to simpler usage in the rest of the extension

dzbarsky · 2025-09-30T00:40:54Z

pip/private/whl_install/repository.bzl

+load("//pip/private/constraints/platform:defs.bzl", "supported_platform")
+load(":parse_whl_name.bzl", "parse_whl_name")
+
+def format_arms(d):


https://bazelbuild.slack.com/archives/CDCMRLS23/p1757528874106669

do with this as you will :)

dzbarsky · 2025-09-30T00:41:30Z

pip/private/whl_install/repository.bzl

+load(":parse_whl_name.bzl", "parse_whl_name")
+
+def format_arms(d):
+    content = ["        \"{}\": \"{}\"".format(k, v) for k, v in d.items()]


maybe clearer to use single quotes on the outside and/or repr the k/v to avoid inner ones?

dzbarsky · 2025-09-30T00:42:08Z

pip/private/whl_install/repository.bzl

+def sort_select_arms(arms):
+    # {(python, platform, abi): target}
+    pairs = list(arms.items())
+    pairs = sorted(pairs, key=select_key, reverse=True)


can you pass arms.items() directly into sorted and skip the copy?

pip/private/whl_install/rule.bzl

dzbarsky · 2025-09-30T00:44:41Z

pip/private/whl_install/rule.bzl

+    return [
+        # FIXME: Need to generate PyInfo here
+        DefaultInfo(
+            files = depset([


i guess you could create this depset once and reuse it below

arrdem added 8 commits August 13, 2025 16:37

[NO TESTS] WIP

8ecc4f7

[NO TESTS] WIP

869c736

[NO TESTS] WIP

33f5861

[NO TESTS] WIP

f1677e3

wip

be8e869

wip

50fe90b

[NO TESTS] WIP

b33ffb6

Merge remote-tracking branch 'origin/main' into arrdem/feat-pip

c12d501

arrdem added 6 commits September 25, 2025 17:02

wip

ae54cb6

Hub sorta builds

cb05389

Got select sorta working

f18274f

Wiring up an example

39f8053

[NO TESTS] WIP

cde1870

Working example

0ce4ea9

Still need to plumb in toolchains & real installer.

dzbarsky reviewed Sep 27, 2025

View reviewed changes

pip/private/sdist_build/rule.bzl Outdated Show resolved Hide resolved

arrdem added 2 commits September 29, 2025 16:39

Multiple venvs working

5fc76f1

wip

7119541

dzbarsky reviewed Sep 29, 2025

View reviewed changes

dzbarsky reviewed Sep 30, 2025

View reviewed changes

arrdem added 6 commits September 30, 2025 23:21

[NO TESTS] WIP

ea3d238

[NO TESTS] WIP

23f28bc

[NO TESTS] WIP

87fb65b

Cutting over to prerelease toml tool

65f64b0

Vendor in the tomltool library in lieu of BCR release

ef98024

Finish adding transition attrs.

6b24ccd

arrdem marked this pull request as ready for review October 3, 2025 17:36

arrdem added 3 commits October 3, 2025 12:04

Start addressing lint.

63854cb

More lint.

f2edc08

Jimmy: Match the rules_python style @pip//cowsay labels

35f7e53

arrdem added 5 commits October 3, 2025 13:44

Jimmy: Integrate the code to match rules_python's normalization

e714a39

Working on lint & docs.

2e46a34

Fix.

f175cf7

Update bzl_library graph

1ce369d

Add a docfile which we need to populate

db91365

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

feat(pip): Add a clean-sheet pip implementation #650

feat(pip): Add a clean-sheet pip implementation #650

Uh oh!

arrdem commented Sep 25, 2025 •

edited

Loading

Uh oh!

aspect-workflows bot commented Sep 25, 2025 •

edited

Loading

Uh oh!

Uh oh!

dzbarsky Sep 29, 2025

Uh oh!

dzbarsky left a comment

Uh oh!

dzbarsky Sep 29, 2025

Uh oh!

dzbarsky Sep 29, 2025

Uh oh!

dzbarsky Sep 30, 2025

Uh oh!

dzbarsky Sep 30, 2025

Uh oh!

dzbarsky Sep 30, 2025

Uh oh!

dzbarsky Sep 30, 2025

Uh oh!

dzbarsky Sep 30, 2025

Uh oh!

dzbarsky Sep 30, 2025

Uh oh!

Uh oh!

dzbarsky Sep 30, 2025

Uh oh!

Uh oh!

Uh oh!

feat(pip): Add a clean-sheet pip implementation #650

Are you sure you want to change the base?

feat(pip): Add a clean-sheet pip implementation #650

Uh oh!

Conversation

arrdem commented Sep 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Features

Example

Appendix

Changes are visible to end-users: yes

To do list

Test plan

Uh oh!

aspect-workflows bot commented Sep 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dzbarsky left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

arrdem commented Sep 25, 2025 •

edited

Loading

aspect-workflows bot commented Sep 25, 2025 •

edited

Loading